Reinforcement learning of dynamic motor sequence: learning to stand up

I n this paper, we propose a learning method f o r implementing human-like sequential movements in robots. As an example of dynamic sequential movement, we consider the “stand-up” task f o r a two-joint, three-link robot. In contrast t o the case of steady walking or standing, the desired trajectory fo r such a transient behavior is very dificult t o derive. The goal of the task is to find a path that links a lying state to a n upright state under the constraints of the system dynamics. The geometry of the robot i s such that there is no static solution; the robot has to stand up dynamically utilizing the momentum of its body. W e use reinforcement learning, in particular, a conitinuous t ime and state temporal difference (TO) learning method. For successful results, we use 1) a n eficient method of value function approximation in a high-dimensional state space, and 2) a hierarchical architecture which divides a large state space into a few smaller pieces.